Fall 2025 Capstone for Data Science
2025-11-28
\[ \text{Obj} = \sum_{i=1}^{n} L(y_i, \hat{y}_i) + \sum_{k=1}^{K} \Omega(f_k) \]
To facilitate analysis, all categorical variables in the dataset were transformed into numerical or grouped categorical formats in Excel. The transformations were applied as follows:
VisulizationTuningResults
The select_best() function from the tidymodels framework was used to systematically extract the hyperparameter combination that achieved the highest ROC AUC across all resampled evaluations.
The resulting optimal configuration included 3 predictors (mtry = 3), 627 boosting iterations, a minimum node size (min_n) of 2, a learning rate of 0.16501, a loss-reduction parameter of 0.0034, and a subsample proportion of 0.5655.
BestParams
Prediction
The model’s predictions on the held-out test set showed strong agreement with the expert diagnoses. Specifically:
The XGBoost multiclass classification model demonstrates moderate overall predictive performance, with room for improvement but also several encouraging indicators of discriminative ability. - Accuracy: 0.654
An accuracy of 65.4% suggests that the model correctly classifies roughly two-thirds of the observations. - Kappa: 0.534
The Kappa value of 0.534 reflects moderate agreement between the model’s predictions and the true class labels after accounting for chance. - ROC AUC (macro-weighted): 0.927
The macro-weighted ROC AUC of 0.927 suggests that the model does a strong job of ranking observations correctly across all classes, even if it occasionally makes incorrect final class assignments.
Taken together, the metrics indicate that the model has strong underlying discriminative power (high AUC) but moderate final classification performance (accuracy and kappa).
A SHAP values analysis was performed to highlight how predictors influence classification across four classes, revealing both broadly influential and class-specific features.
Most Influential Features (Across Classes)
Class-Specific Influences
Overall Conclusion
The final XGBoost model delivered: - High discriminative performance, with ROC AUC > 0.90 across cross-validation and sensitivity analyses
- Stable, robust generalization, supported by low performance variability
- Interpretable, clinically relevant insights, enabled by SHAP and feature importance
- Accurate classification, with strong performance across all four diagnostic groups on unseen data
Overall, XGBoost proved to be an effective and interpretable modeling approach for mental health classification in a small structured dataset, delivering both predictive value and meaningful interpretability essential for clinical research.